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PURPOSE OF DECLARATION 

1 . This declaration is to establish reduction to practice of the claimed invention of 
this application in the United States at a date prior to February 19, 2003, which is the effective 
date of the primary reference Jarman et al, U.S. Patent Pub. No. 2004/0184670 cited by the 
Examiner. 

2. The person making this declaration is the inventor, A. Mufit Ferman. 

FACTS AND DOCUMENTARY EVIDENCE 

3. A working prototype of the inventions claimed in the captioned application was 
constructed on a date prior to February 19, 2003. Attached to this declaration is an Exhibit 
showing an invention disclosure form, signed by me on a date also prior to February 19 2003, 
documenting the existence of a working prototype. 

4. The date by which the prototype described in paragraph 3 was constructed is prior 
the effective date of the cited primary reference, Jarman. 

5. This declaration is submitted concurrently with the filing of an RCE. 



DECLARATION 



6. As a person signing below, I hereby declare that all statements made herein of my 
own knowledge are true and that all statements made on information and belief are believed to be 
true; and further that these statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or imprisonment, or both, under Section 
1001 of Title 18 of the United States Code, and that such willful false statements may jeopardize 
the validity of the application or any patent issued thereon. 

SIGNATURE 

<^7zZ.^ l/ZS/of 
A. Mufit Ferman Date 



Page 2 

C:\Documentt and Settings\dripma\Local ScttingsXTcmporary Internet Files\OLK7EVDeclaratioii.doc 




SHARP Laboratories of America, Inc. 

5750 NW Pacific Rim Boulevard! 1 
Camas, Washington 98607 



SLA Confidential 



Page 1 of 1 
SLA Docket No. 



INVENTION DISCLOSURE FORM 

Revised 4/02 

Has a provisional patent application or related patent application been filed previously for this 
case or a closely related case? If so, indicate SLA numbers of previously filed cases. 



SLA 


Relation, (i.e., provisional, continuation-in-part, or just technically related) 


SLA 


Relation: ' " 


SLA 


Relation: 


SLA 


Relation: 



2. Descriptive Title of Invention: 

Method and System for Automatic Detection of Red-Eye Artifacts in Digital Color Photos 



3. Inventor(s): 



Full Legal Name: 


Ahmet 


Mufit 




Ferman 




First 


Middle 




Last 


Address: 


3426 SE 170 th Ave 


Vancouver 


WA 


98683 Clark 


Citizenship: 


Street 


City 


State 


Zip County I 




Turkey 




Telephone: 


360 260 3543 


360 817 7557 








Home 


Company 








Full Legal Name: 












First 


Middle 




Last 


Address: 










Citizenship: 


Street 


City 


State 


Zip County 








Telephone: 












Home 


Company 















Inventor Signature 


Date 
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Full Legal Name: 




First 


Middle 




Last 


Address: 










Citizenship: 


Street 


City 


State 


Zip County 


Telephone: 












Home 


Company 







Supervisor's Acknowledgment: 


"1 believe this disclosure is novel and complete and should 
be submitted to the Patent Review Committee." 


Supervisor's Signature: 
Date: 




Supporting Group: □ CRDG □ AVSG □ CSG □ ISG |S|DSG 


□ ICG 


□ LCDG □ Other: 


Inventors' Department: □ DV 


□ mmc KIdi Dist Die Dlcd 



4. Project & Supervisor: 

Supervisor's Name: 
Supervisor's Title: 
Project Number/Name: 



Michael A. Kriss 



Manager, Color & Imaging Group 



IUIP06 - Object Identification & ROI 



5. Conception of the Invention: 

Date Conceived: 
Date of first Written Description: 
Notebook & Page No. or File Archive: 
Date first explained to others (whom?): 
Planned Application for the Invention: 



Inventor Signature 



Date 



Inventor Signature 



Date 



Inventor Signature 



Date 



Approx.* 



Printer, copier, scanner and digital camera image 
pipelines; document/image management and 
manipulation software 




- -—Witnessed &iJnjJerstood By 



Witnessed & Understood By 




4:55 PM February 7, 2003 



Invention Disclosure 



RedEyeDisc 



SHARP Laboratories of Arp^rica, Inc. 

5750 NW Pacific Rim Boulevarct j 
Camas, Washington 98607 



SLA Confident^ p age 3 of 1 9 

SLA Docket No, 



6. Construction & Test of First Prototype Embodying the Invention 

Date First Prototype Completed: 

Part Number/Product Description: 

Date of First Successful Test: 

Successful Operation Witnessed By: 



7. Public Disclosure of Invention (Presentation at public meeting or publication 
(NOTE: Patent Application MUST be filed prior to any public disclosure.). 

Date of First Public Disclosure: None 

Setting (Conference/Journal Name): 
Title of Paper or Presentation : 
Type of Disclosure (Written/Verbal): 



Does Data Sheet or Application Note Disclose the Invention (when)? 



8. Would one skilled in the art of your invention need to refer to any published documents, in 
addition to your disclosure, in order to make and use your invention? If so, list references and 
summarize background material therein. 

No additional references are required for a person skilled in the art of this invention to understand, 
make, and use the proposed method. 

9. Briefly summarize the primary unique aspects of your invention and the problems solved by the 
invention. 

Red-eye is a common phenomenon that occurs during flash photography. In an environment where a 
flash is needed to illuminate the subject, the subject's pupils are dilated due to the low ambient 
illumination. Light from the flash can then enter the eye through the pupil and be reflected off the 
blood vessels at the back of the retina. This reflection is recorded by the camera if the geometry of 
the camera lens, the flash, and the subject's eye is just right, rendering the captured image 
unpleasant and objectionable to viewers. Hence there is a significant need for automatic methods that 
identify and correct red-eye areas in a captured image. 

A number of methods have been proposed in the prior art for detecting and/or removing red-eye 
artifacts. Most of these methods are either (i) supervised; i.e. they require the user to manually 
identify the subregions in an image where the artifacts are observed, or>(ii) dependent on skin/face 
and/or eye detection to find the areas of interest. In this invention we propose an unsupervised red- 
eye detection technique that uses low-level image features to locate the red-eye pixels in a digital 
image. The algorithm does not require detection of the face and/or skin regions in an image, and is 
therefore significantly different from the prior art. Another advantage of the invention is that it limits 
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processing to those areas in the image that are affected by the flash illumination, since the red-eye 
phenomenon is a direct result of flash use. The computational overhead is therefore reduced 
considerably. The algorithm utilizes low-level image features and basic processing techniques such 
as median filtering and morphological operations, as well as color- and shape-based constraints, 
repeatedly and in succession to reduce the number of candidate regions that may correspond to red- 
eye. Furthermore, each component of the image color space is analyzed in a specific way, and the 
results are finally merged to yield a more reliable output. The invention can be implemented as a 
stand-alone computer application that operates on digital images or as a plug-in to other 
image/document management software (e.g., SharpDesk, Photoshop); or it may be incorporated into 
an MFP. 

10. Detailed Description. 

Fig. 1 illustrates an embodiment of the invention. The input to the system is a color digital image, which 
may be in any one of a number of difference color spaces. In the current embodiment, the input image is 
converted into the HSV (hue/saturation/value) or HSI (hue/saturation/intensity) color space, where the 
rest of the processing occurs. Specifically, each channel of the HSV color space is processed and 
analyzed in various ways to accurately identify the red-eye artifacts. 

As discussed in the previous section, red-eye artifacts occur as a direct consequence of flash. The red- 
eye detection method therefore focuses only on those regions of the image that have been affected (i.e. 
illuminated) by a flash. To identify these regions a thresholding operation is first applied to the brightness 
(V) component I v of the image. The pixels that exceed the threshold value 2}comprise the flash mask, M fl : 



The value of threshold 7)can be selected in various ways. In the current embodiment, 7} is computed for 
each input image individually using Otsu's thresholding method (Otsu, N. (1979), A thresholding selection 
method from gray-level histogram, in IEEE Trans. Syst Man Cybernet 9(1), 62-66.); however, other 
threshold selection methods can also be used. Furthermore, the value of 7} can be used to determine 
whether the input image is a flash image or not. 

Once the initial flash mask is obtained, several post-processing operations can be applied to refine it by 
eliminating isolated pixels. These operations may include, but are not limited to, median filtering, 
morphological operations such as erosion and opening, and so on. The remaining pixels in M f are then 
grouped into contiguous regions using a connected component labeling algorithm, and the areas of the 
labeled regions are computed. Regions with areas smaller than a predetermined threshold are discarded. 
The convex hull of each remaining region is subsequently computed and a binary mask that comprises 
the union of the convex hulls is constructed to yield the final flash mask A/^. Figure 2 highlights the various 
stages in the construction of M f . Fig. 2(a) depicts the input image /; the V component of the image, I V1 is 
shown in 2(b). The results of the thresholding and post-processing operations are shown in Figs. 2(c) and 
(d), respectively. The final flash mask M fl obtained after area-based thresholding and convex hull 
generation, is depicted in 2(e). A/^ represents the areas in the input image that may contain red-eye 
artifacts; therefore, the rest of the processing is restricted to the regions identified by M f . 



After M f \s computed, it is used to perform further processing on the hue component I h . We first apply M i 
to I h and obtain a masked version 7 A m . Hue corresponds to the dominant color of a pixel, and it is 
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represented as an angle on the unit circle between 0° and 360°. When the hue values are mapped to an 
appropriate interval for display (e.g., to [0,1] or [0,255]), red-eye locations are observed to always appear 
as light, contiguous regions on darker backgrounds, as shown in Fig. 3(a). We exploit this property by 

thresholding /" to eliminate the dark areas and thus reduce the area that needs to be analyzed for red- 



The value of the threshold T h can be chosen in different ways. In the current embodiment, T h e [0,1], and 
is set to 0.125. 

After M h is obtained, several post-processing operations may be applied to refine it. These operations 
may include, but are not limited to, median filtering, morphological filtering such as dilation and closing, 
and so on. The selected pixels in M h are then grouped into contiguous regions using a connected 
component labeling algorithm, and several features are computed for each labeled region. Specifically, 
we consider the area, aspect ratio, and extent of every region to determine the likelihood that the region is 
a red-eye area. Extent is defined as the ratio of the total area of the region (i.e. the number of pixels in the 
region) to the number of pixels in the smallest bounding box for the region. Regions whose areas and/or 
aspect ratios fall outside predetermined ranges, or whose extent values are below a specified threshold, 
are discarded. In the current embodiment, the minimum and maximum allowed sizes for a region are 
computed dynamically based on the size of the input image. The aspect ratio test allows us to eliminate 
regions that are elongated; the aspect ratio of a candidate red-eye region is expected to be in the interval 
(0.33,2). Finally, if the extent of a region is less than 0.33, the region is removed from the list of candidate 
red-eye locations. 

Figure 3 highlights the various stages in the construction of M h . Fig. 3(a) depicts the hue component I h of 
the image; the masked hue component, /* , is depicted in 3(b). The result of the thresholding and post- 
processing operations is shown in Fig. 3(c). The final hue maskM A> obtained after connected component 
labeling and area- and shape-based filtering is depicted in 3(d). 

Next, we utilize the information in the saturation component of the image to further refine the list of 
candidate red-eye regions. It is observed that pixels in the red-eye regions often have high saturation 
values, as seen in the example image in Fig. 2(a). This phenomenon is also clearly demonstrated in Fig. 
4(a), which shows the saturation component I s for the example image. Furthermore, the local variation in 
the saturation component is highly pronounced around the red-eye regions. To exploit this property we 
then compute the standard deviation of the saturation component for each pixel using a local 
neighborhood (Fig. 4(b)). Pixels that are likely to be red-eye artifacts are then identified by a thresholding 
operation, which yields the saturation mask M C1 as shown in Fig. 4(c). The value of the threshold can be 
chosen in different ways. In the current embodiment, it is set to 0.15 based on empirical evidence. 

The intersection of M h and M 9 is then computed to yield a final mask M ha (Fig. 4(d)) that represents the 
locations where the red-eye artifacts are most likely to occur. As in earlier stages of the algorithm, several 
post-processing operations may be applied to refine M ha . These operations may include, but are not 
limited to, median filtering, morphological filtering such as dilation and closing, and so on. The selected 
pixels in M ho are then grouped into contiguous regions using a connected component labeling algorithm, 
and several shape-based features are computed for each labeled region. Specifically, we compute the 











Inventor Signature 


Date 






Inventor Signature 


Date 


Witnessed & Understood By 


Date 


Inventor Signature 


Date 


Witnessed & Understood By 


Date 



eye artifacts: 




4:47 PM September 22, 2003 



Invention Disclosure 



RedEyeDisc 



.SHARP Laboratories of Africa, Inc. SLA Confidential p age 6 of 1 9 

5750 NW Pacific Rim Boulevarcl ) I ] 

Camas, Washington 98607 SLA Docket No. 



eccentricity and circularity of every region. Eccentricity is defined as the ratio of the distance between the 
foci of the ellipse that has the same second-moments as the region and its major axis length. The value 
of eccentricity varies between 0 and 1 ; the higher the eccentricity value, the closer to a line segment the 
region is. Circularity, as the name implies, is a measure of how closely a region resembles a circle, and is 
defined as the ratio of the square of the region perimeter to the area of the region. These properties are 
used to determine the likelihood that a particular region contains red-eye artifacts (Fig. 4(e)). The final 
stage of the algorithm involves color-based analysis of the remaining regions to determine which pixels 
are strongly red. This can be achieved using the hue component, by specifying the appropriate range of 
hue angles corresponding to color red. Alternatively this color test can be carried out in other color 
spaces, such as RGB, YCrCb, La*b*, and so on. In the current embodiment, we utilize the RGB values of 
the pixels in each candidate region to determine whether the region contains a red-eye artifact. The RGB 
values can be computed directly from the available HSV components be means of a simple 
transformation. For every region, we compute the mean of each primary. We then observe whether (i) the 
mean red value is less than a specified threshold, or (ii) the ratio of the means of the green and blue 
components is below another predetermined threshold. Any region that satisfies either of the above 
criteria is discarded, and the remaining regions are declared red-eye artifact locations (Fig. 4(f)). The 
individual pixels that require correction within these regions are identified through an analysis of the color 
properties of the individual pixels. This analysis may include, but is not limited to, thresholding based on 
pixel color values, clustering/region growing based on color similarity, and so on. The final output of the 
algorithm is a mask that identifies the individual pixels in the image that require red-eye correction. 
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Input Image 




Locations 



Figure 1 Flowchart of the automatic red-eye detection system 
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Figure 2 Construction of the flash mask for the input image using the brightness component: (a) Original 
input image; (b) brightness component /„; (c) flash mask If obtained after thresholding by T h \ (d) Rafter 
post-processing to eliminate isolated pixels, and (e) final mask image Afy obtained after connected 
component labeling, area-based thresholding and convex hull generation. 
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Figure 3 Identification of candidate red-eye regions using My and the hue component: (a) Hue component 
4; (b) 7™ obtained after masking I h by M/, (c) remaining regions in 7™ after thresholding and post- 
processing operations; and (d) final set of candidate red-eye artifact locations after connected component 
labeling and area- and shape-based filtering. (The unprocessed background regions are shown in blue.) 
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(e) (f) 



Figure 4 Identification of the red-eye artifact regions using the saturation component: (a) Saturation 
component (b) standard deviation map I s<r \ (c) mask obtained by thresholding / Jff ; (d) candidate red-eye 
regions obtained by intersection of M h and M„ and after (e) shape-based filtering, and (f) final red-eye 
locations identified after color-based analysis. The pixels in these regions are further inspected 
individually to determine those that require correction. (The unprocessed background regions are shown 
in blue.) 
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